Influence of training and expertise on deep neural network attention and human attention during a medical image classification task.

In many different domains, experts can make complex decisions after glancing very briefly at an image. However, the perceptual mechanisms underlying expert performance are still largely unknown. Recently, several machine learning algorithms have been shown to outperform human experts in specific tasks. But these algorithms often behave as black boxes and their information processing pipeline remains unknown. This lack of transparency and interpretability is highly problematic in applications involving human lives, such as health care. One way to "open the black box" is to compute an artificial attention map from the model, which highlights the pixels of the input image that contributed the most to the model decision. In this work, we directly compare human visual attention to machine visual attention when performing the same visual task. We have designed a medical diagnosis task involving the detection of lesions in small bowel endoscopic images. We collected eye movements from novices and gastroenterologist experts while they classified medical images according to their relevance for Crohn's disease diagnosis. We trained three state-of-the-art deep learning models on our carefully labeled dataset. Both humans and machine performed the same task. We extracted artificial attention with six different post hoc methods. We show that the model attention maps are significantly closer to human expert attention maps than to novices', especially for pathological images. As the model gets trained and its performance gets closer to the human experts, the similarity between model and human attention increases. Through the understanding of the similarities between the visual decision-making process of human experts and deep neural networks, we hope to inform both the training of new doctors and the architecture of new algorithms.

Shorter self-reported sleep duration is associated with worse virtual spatial navigation performance in men.

Sleep has been shown to impact navigation ability. However, it remains unclear how different sleep-related variables may be independently associated with spatial navigation performance, and as to whether gender may play a role in these associations. We used a mobile video game app, Sea Hero Quest (SHQ), to measure wayfinding ability in US-based participants. Wayfinding performance on SHQ has been shown to correlate with real-world wayfinding. Participants were asked to report their sleep duration, quality, daytime sleepiness and nap frequency and duration on a typical night (n = 766, 335 men, 431 women, mean age = 26.5 years, range = 18-59 years). A multiple linear regression was used to identify which self-reported sleep variables were independently associated with wayfinding performance. Shorter self-reported sleep durations were significantly associated with worse wayfinding performance in men only. Other self-reported sleep variables showed non-significant trends of association with wayfinding performance. When removing non-typical sleepers (< 6 or > 9 h of sleep on a typical night), the significant association between sleep duration and spatial navigation performance in men was no longer present. These findings from U.S.-based participants suggest that a longer self-reported sleep duration may be an important contributor to successful navigation ability in men.

Influence of prior knowledge on eye movements to scenes as revealed by hidden Markov models.

Human visual experience usually provides ample opportunity to accumulate knowledge about events unfolding in the environment. In typical scene perception experiments, however, participants view images that are unrelated to each other and, therefore, they cannot accumulate knowledge relevant to the upcoming visual input. Consequently, the influence of such knowledge on how this input is processed remains underexplored. Here, we investigated this influence in the context of gaze control. We used sequences of static film frames arranged in a way that allowed us to compare eye movements to identical frames between two groups: a group that accumulated prior knowledge relevant to the situations depicted in these frames and a group that did not. We used a machine learning approach based on hidden Markov models fitted to individual scanpaths to demonstrate that the gaze patterns from the two groups differed systematically and, thereby, showed that recently accumulated prior knowledge contributes to gaze control. Next, we leveraged the interpretability of hidden Markov models to characterize these differences. Additionally, we report two unexpected and interesting caveats of our approach. Overall, our results highlight the importance of recently acquired prior knowledge for oculomotor control and the potential of hidden Markov models as a tool for investigating it.

Examining individual learning patterns using generalised linear mixed models.

Everyone learns differently, but individual performance is often ignored in favour of a group-level analysis. Using data from four different experiments, we show that generalised linear mixed models (GLMMs) and extensions can be used to examine individual learning patterns. Producing ellipsoids and cluster analyses based on predicted random effects, individual learning patterns can be identified, clustered and used for comparisons across various experimental conditions or groups. This analysis can handle a range of datasets including discrete, continuous, censored and non-censored, as well as different experimental conditions, sample sizes and trial numbers. Using this approach, we show that learning a face-named paired associative task produced individuals that can learn quickly, with the performance of some remaining high, but with a drop-off in others, whereas other individuals show poor performance throughout the learning period. We see this more clearly in a virtual navigation spatial learning task (NavWell). Two prominent clusters of learning emerged, one showing individuals who produced a rapid learning and another showing a slow and gradual learning pattern. Using data from another spatial learning task (Sea Hero Quest), we show that individuals' performance generally reflects their age category, but not always. Overall, using this analytical approach may help practitioners in education and medicine to identify those individuals who might need extra help and attention. In addition, identifying learning patterns may enable further investigation of the underlying neural, biological, environmental and other factors associated with these individuals.

Prior knowledge about events depicted in scenes decreases oculomotor exploration.

The visual input that the eyes receive usually contains temporally continuous information about unfolding events. Therefore, humans can accumulate knowledge about their current environment. Typical studies on scene perception, however, involve presenting multiple unrelated images and thereby render this accumulation unnecessary. Our study, instead, facilitated it and explored its effects. Specifically, we investigated how recently-accumulated prior knowledge affects gaze behavior. Participants viewed sequences of static film frames that contained several 'context frames' followed by a 'critical frame'. The context frames showed either events from which the situation depicted in the critical frame naturally followed, or events unrelated to this situation. Therefore, participants viewed identical critical frames while possessing prior knowledge that was either relevant or irrelevant to the frames' content. In the former case, participants' gaze behavior was slightly more exploratory, as revealed by seven gaze characteristics we analyzed. This result demonstrates that recently-gained prior knowledge reduces exploratory eye movements.

Wayfinding and path integration deficits detected using a virtual reality mobile app in patients with traumatic brain injury.

The ability to navigate is supported by a wide network of brain areas which are particularly vulnerable to disruption brain injury, including traumatic brain injury (TBI). Wayfinding and the ability to orient back to the direction you have recently come (path integration) may likely be impacted in daily life but have so far not been tested with patients with TBI. Here, we assessed spatial navigation in thirty-eight participants, fifteen of whom had a history of TBI, and twenty-three control participants. Self-estimated spatial navigation ability was assessed using the Santa Barbara Sense of Direction (SBSOD) scale. No significant difference between TBI patients and a control group was identified. Rather, results indicated that both participant groups demonstrated 'good' self-inferred spatial navigational ability on the SBSOD scale. Objective navigation ability was tested via the virtual mobile app test Sea Hero Quest (SHQ), which has been shown to predict real-world navigation difficulties and assesses (a) wayfinding across several environments and (b) path integration. Compared to a sub-sample of 13 control participants, a matched subsample of 10 TBI patients demonstrated generally poorer performance on all wayfinding environments tested. Further analysis revealed that TBI participants consistently spent a shorter duration viewing a map prior to navigating to goals. Patients showed mixed performance on the path integration task, with poor performance evident when proximal cues were absent. Our results provide preliminary evidence that TBI impacts both wayfinding and, to some extent, path integration. The findings suggest long-lasting clinical difficulties experienced in TBI patients affect both wayfinding and to some degree path integration ability.

Landmark-dependent Navigation Strategy Declines across the Human Life-Span: Evidence from Over 37,000 Participants.

Humans show a remarkable capacity to navigate various environments using different navigation strategies, and we know that strategy changes across the life span. However, this observation has been based on studies of small sample sizes. To this end, we used a mobile app-based video game (Sea Hero Quest) to test virtual navigation strategies and memory performance within a distinct radial arm maze level in over 37,000 participants. Players were presented with six pathways (three open and three closed) and were required to navigate to the three open pathways to collect a target. Next, all six pathways were made available and the player was required to visit the pathways that were previously unavailable. Both reference memory and working memory errors were calculated. Crucially, at the end of the level, the player was asked a multiple-choice question about how they found the targets (i.e., a counting-dependent strategy vs. a landmark-dependent strategy). As predicted from previous laboratory studies, we found the use of landmarks declined linearly with age. Those using landmark-based strategies also performed better on reference memory than those using a counting-based strategy. These results extend previous observations in the laboratory showing a decreased use of landmark-dependent strategies with age.

Explaining World-Wide Variation in Navigation Ability from Millions of People: Citizen Science Project Sea Hero Quest.

Navigation ability varies widely across humans. Prior studies have reported that being younger and a male has an advantage for navigation ability. However, these studies have generally involved small numbers of participants from a handful of western countries. Here, we review findings from our project Sea Hero Quest, which used a video game for mobile and tablet devices to test 3.9 million people on their navigation ability, sampling across every nation-state and from 18 to 99 years of age. Results revealed that the task has good ecological validity and across all countries sufficiently sampled (N = 63), age is linked to a near-linear decline in navigation ability from the early 20s. All countries showed a male advantage, but this varied considerably and could be partly predicted by gender inequality. We found that those who reported growing up in a city were on average worse at navigating than those who grew up outside cities and that navigation performance helped identify those at greater genetic risk of Alzheimer's disease. We discuss the advantages and challenges of using a mobile app to study cognition and the future avenues for understanding individual differences in navigation ability arising from this research.

Communicating and reading emotion with masked faces in the Covid era: A short review of the literature.

Face masks have proven to be key to slowing down the SARS-Cov2 virus spread in the COVID-19 pandemic context. However, wearing face masks is not devoid of "side-effects", at both the physical and psychosocial levels. In particular, masks hinder emotion reading from facial expressions as they hide a significant part of the face. This disturbs both holistic and featural processing of facial expressions and, therefore, impairs emotion recognition, and influences many aspects of human social behavior. Communication in general is disrupted by face masks, as they modify the wearer's voice and prevent the audience from using lip reading or other non-verbal cues for speech comprehension. Individuals suffering from psychiatric conditions with impairment of communication, are at higher risk of distress because masks increase their difficulties to read emotions from faces. The identification and acknowledgement of these "side-effects" on communication are necessary because they warrant further work on adaptive solutions that will help foster the use of face masks by the greatest number.

Does culture shape our understanding of others' thoughts and emotions? An investigation across 12 countries.

Measures of social cognition have now become central in neuropsychology, being essential for early and differential diagnoses, follow-up, and rehabilitation in a wide range of conditions. With the scientific world becoming increasingly interconnected, international neuropsychological and medical collaborations are burgeoning to tackle the global challenges that are mental health conditions. These initiatives commonly merge data across a diversity of populations and countries, while ignoring their specificity. OBJECTIVE: In this context, we aimed to estimate the influence of participants' nationality on social cognition evaluation. This issue is of particular importance as most cognitive tasks are developed in highly specific contexts, not representative of that encountered by the world's population. METHOD: Through a large international study across 18 sites, neuropsychologists assessed core aspects of social cognition in 587 participants from 12 countries using traditional and widely used tasks. RESULTS: Age, gender, and education were found to impact measures of mentalizing and emotion recognition. After controlling for these factors, differences between countries accounted for more than 20% of the variance on both measures. Importantly, it was possible to isolate participants' nationality from potential translation issues, which classically constitute a major limitation. CONCLUSIONS: Overall, these findings highlight the need for important methodological shifts to better represent social cognition in both fundamental research and clinical practice, especially within emerging international networks and consortia. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Multi-expert annotation of Crohn's disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network.

Background and study aims Computer-aided diagnostic tools using deep neural networks are efficient for detection of lesions in endoscopy but require a huge number of images. The impact of the quality of annotation has not been tested yet. Here we describe a multi-expert annotated dataset of images extracted from capsules from Crohn's disease patients and the impact of the quality of annotations on the accuracy of a recurrent attention neural network. Methods Images of capsule were annotated by a reader first and then reviewed by three experts in inflammatory bowel disease. Concordance analysis between experts was evaluated by Fleiss' kappa and all the discordant images were, again, read by all the endoscopists to obtain a consensus annotation. A recurrent attention neural network developed for the study was tested before and after the consensus annotation. Available neural networks (ResNet and VGGNet) were also tested under the same conditions. Results The final dataset included 3498 images with 2124 non-pathological (60.7 %), 1360 pathological (38.9 %), and 14 (0.4 %) inconclusive. Agreement of the experts was good for distinguishing pathological and non-pathological images with a kappa of 0.79 ( P  < 0.0001). The accuracy of our classifier and the available neural networks increased after the consensus annotation with a precision of 93.7 %, sensitivity of 93 %, and specificity of 95 %. Conclusions The accuracy of the neural network increased with improved annotations, suggesting that the number of images needed for the development of these systems could be diminished using a well-designed dataset.

In the eye of the beholder? Oxytocin effects on eye movements in schizophrenia.

BACKGROUND: Individuals with schizophrenia have difficulty in extracting salient information from faces. Eye-tracking studies have reported that these individuals demonstrate reduced exploratory viewing behaviour (i.e. reduced number of fixations and shorter scan paths) compared to healthy controls. Oxytocin has previously been demonstrated to exert pro-social effects and modulate eye gaze during face exploration. In this study, we tested whether oxytocin has an effect on visual attention in patients with schizophrenia. METHODS: Nineteen male participants with schizophrenia received intranasal oxytocin 40UI or placebo in a double-blind, placebo-controlled, crossover fashion during two visits separated by seven days. They engaged in a free-viewing eye-tracking task, exploring images of Caucasian men displaying angry, happy, and neutral emotional expressions; and control images of animate and inanimate stimuli. Eye-tracking parameters included: total number of fixations, mean duration of fixations, dispersion, and saccade amplitudes. RESULTS: We found a main effect of treatment, whereby oxytocin increased the total number of fixations, dispersion, and saccade amplitudes, while decreasing the duration of fixations compared to placebo. This effect, however, was non-specific to facial stimuli. When restricting the analysis to facial images only, we found the same effect. In addition, oxytocin modulated fixation rates in the eye and nasion regions. DISCUSSION: This is the first study to explore the effects of oxytocin on eye gaze in schizophrenia. Oxytocin had enhanced exploratory viewing behaviour in response to both facial and inanimate control stimuli. We suggest that the acute administration of intranasal oxytocin may have the potential to enhance visual attention in schizophrenia.

Toward personalized cognitive diagnostics of at-genetic-risk Alzheimer's disease.

Spatial navigation is emerging as a critical factor in identifying preclinical Alzheimer's disease (AD). However, the impact of interindividual navigation ability and demographic risk factors (e.g., APOE, age, and sex) on spatial navigation make it difficult to identify persons "at high risk" of AD in the preclinical stages. In the current study, we use spatial navigation big data (n = 27,108) from the Sea Hero Quest (SHQ) game to overcome these challenges by investigating whether big data can be used to benchmark a highly phenotyped healthy aging laboratory cohort into high- vs. low-risk persons based on their genetic (APOE) and demographic (sex, age, and educational attainment) risk factors. Our results replicate previous findings in APOE ε4 carriers, indicative of grid cell coding errors in the entorhinal cortex, the initial brain region affected by AD pathophysiology. We also show that although baseline navigation ability differs between men and women, sex does not interact with the APOE genotype to influence the manifestation of AD-related spatial disturbance. Most importantly, we demonstrate that such high-risk preclinical cases can be reliably distinguished from low-risk participants using big-data spatial navigation benchmarks. By contrast, participants were undistinguishable on neuropsychological episodic memory tests. Taken together, we present evidence to suggest that, in the future, SHQ normative benchmark data can be used to more accurately classify spatial impairments in at-high-risk of AD healthy participants at a more individual level, therefore providing the steppingstone for individualized diagnostics and outcome measures of cognitive symptoms in preclinical AD.

Virtual navigation tested on a mobile app is predictive of real-world wayfinding navigation performance.

Virtual reality environments presented on tablets and smartphones have potential to aid the early diagnosis of conditions such as Alzheimer's dementia by quantifying impairments in navigation performance. However, it is unclear whether performance on mobile devices can predict navigation errors in the real world. We compared the performance of 49 participants (25 females, 18-35 years old) at wayfinding and path integration tasks designed in our mobile app 'Sea Hero Quest' with their performance at similar tasks in a real-world environment. We first performed this experiment in the streets of London (UK) and replicated it in Paris (France). In both cities, we found a significant correlation between virtual and real-world wayfinding performance and a male advantage in both environments, although smaller in the real world (Cohen's d in the game = 0.89, in the real world = 0.59). Results in London and Paris were highly similar, and controlling for familiarity with video games did not change the results. The strength of the correlation between real world and virtual environment increased with the difficulty of the virtual wayfinding task, indicating that Sea Hero Quest does not merely capture video gaming skills. The fact that the Sea Hero Quest wayfinding task has real-world ecological validity constitutes a step toward controllable, sensitive, safe, low-cost, and easy to administer digital cognitive assessment of navigation ability.

Global Determinants of Navigation Ability.

Human spatial ability is modulated by a number of factors, including age [1-3] and gender [4, 5]. Although a few studies showed that culture influences cognitive strategies [6-13], the interaction between these factors has never been globally assessed as this requires testing millions of people of all ages across many different countries in the world. Since countries vary in their geographical and cultural properties, we predicted that these variations give rise to an organized spatial distribution of cognition at a planetary-wide scale. To test this hypothesis, we developed a mobile-app-based cognitive task, measuring non-verbal spatial navigation ability in more than 2.5 million people and sampling populations in every nation state. We focused on spatial navigation due to its universal requirement across cultures. Using a clustering approach, we find that navigation ability is clustered into five distinct, yet geographically related, groups of countries. Specifically, the economic wealth of a nation was predictive of the average navigation ability of its inhabitants, and gender inequality was predictive of the size of performance difference between males and females. Thus, cognitive abilities, at least for spatial navigation, are clustered according to economic wealth and gender inequalities globally, which has significant implications for cross-cultural studies and multi-center clinical trials using cognitive testing.

Personality Traits Do Not Predict How We Look at Faces.

While personality has typically been considered to influence gaze behaviour, literature relating to the topic is mixed. Previously, we found no evidence of self-reported personality traits on preferred gaze duration between a participant and a person looking at them via a video. In this study, 77 of the original participants answered an in-depth follow-up survey containing a more comprehensive assessment of personality traits (Big Five Inventory) than was initially used, to check whether earlier findings were caused by the personality measure being too coarse. In addition to preferred mutual gaze duration, we also examined two other factors linked to personality traits: number of blinks and total fixation duration in the eye region of observed faces. No significant correlations were found between any of these measures and participant personality traits. We suggest that effects previously reported in the literature may stem from contextual differences or modulation of arousal.

Semantic content outweighs low-level saliency in determining children's and adults' fixation of movies.

To make sense of the visual world, we need to move our eyes to focus regions of interest on the high-resolution fovea. Eye movements, therefore, give us a way to infer mechanisms of visual processing and attention allocation. Here, we examined age-related differences in visual processing by recording eye movements from 37 children (aged 6-14years) and 10 adults while viewing three 5-min dynamic video clips taken from child-friendly movies. The data were analyzed in two complementary ways: (a) gaze based and (b) content based. First, similarity of scanpaths within and across age groups was examined using three different measures of variance (dispersion, clusters, and distance from center). Second, content-based models of fixation were compared to determine which of these provided the best account of our dynamic data. We found that the variance in eye movements decreased as a function of age, suggesting common attentional orienting. Comparison of the different models revealed that a model that relies on faces generally performed better than the other models tested, even for the youngest age group (<10years). However, the best predictor of a given participant's eye movements was the average of all other participants' eye movements both within the same age group and in different age groups. These findings have implications for understanding how children attend to visual information and highlight similarities in viewing strategies across development.

Scanpath modeling and classification with hidden Markov models.

How people look at visual information reveals fundamental information about them; their interests and their states of mind. Previous studies showed that scanpath, i.e., the sequence of eye movements made by an observer exploring a visual stimulus, can be used to infer observer-related (e.g., task at hand) and stimuli-related (e.g., image semantic category) information. However, eye movements are complex signals and many of these studies rely on limited gaze descriptors and bespoke datasets. Here, we provide a turnkey method for scanpath modeling and classification. This method relies on variational hidden Markov models (HMMs) and discriminant analysis (DA). HMMs encapsulate the dynamic and individualistic dimensions of gaze behavior, allowing DA to capture systematic patterns diagnostic of a given class of observers and/or stimuli. We test our approach on two very different datasets. Firstly, we use fixations recorded while viewing 800 static natural scene images, and infer an observer-related characteristic: the task at hand. We achieve an average of 55.9% correct classification rate (chance = 33%). We show that correct classification rates positively correlate with the number of salient regions present in the stimuli. Secondly, we use eye positions recorded while viewing 15 conversational videos, and infer a stimulus-related characteristic: the presence or absence of original soundtrack. We achieve an average 81.2% correct classification rate (chance = 50%). HMMs allow to integrate bottom-up, top-down, and oculomotor influences into a single model of gaze behavior. This synergistic approach between behavior and machine learning will open new avenues for simple quantification of gazing behavior. We release SMAC with HMM, a Matlab toolbox freely available to the community under an open-source license agreement.

Visual Attention Saccadic Models Learn to Emulate Gaze Patterns From Childhood to Adulthood.

How people look at visual information reveals fundamental information about themselves, their interests and their state of mind. While previous visual attention models output static 2D saliency maps, saccadic models aim to predict not only where observers look at but also how they move their eyes to explore the scene. In this paper, we demonstrate that saccadic models are a flexible framework that can be tailored to emulate observer's viewing tendencies. More specifically, we use fixation data from 101 observers split into five age groups (adults, 8-10 y.o., 6-8 y.o., 4-6 y.o., and 2 y.o.) to train our saccadic model for different stages of the development of human visual system. We show that the joint distribution of saccade amplitude and orientation is a visual signature specific to each age group, and can be used to generate age-dependent scan paths. Our age-dependent saccadic model does not only output human-like, age-specific visual scan paths, but also significantly outperforms other state-of-the-art saliency models. We demonstrate that the computational modeling of visual attention, through the use of saccadic model, can be efficiently adapted to emulate the gaze behavior of a specific group of observers.